Content Determination in GRE: Evaluating the evaluator

In this paper, we discuss the evaluation measures proposed in a number of recent papers associated with the TUNA project1, and which have become an important component of the First NLG Shared Task and Evaluation Campaign (STEC) on attribute selection for referring expressions generation. Focusing on reference to individual objects, we discuss what such evaluation measures should be expected to achieve, and what alternative measures merit consideration. The measures mentioned above can be motivated as follows. Suppose a large number of utterance situations had been defined, where each situation contained a number of objects, one of which needed to be described by a referring expression. Suppose, furthermore, one had an infallible oracle which told us, for each of these situations, what was the best referring expression for that situation. How could this oracle be used to evaluate the extent to which other referring expressions are similar to the one proposed by the oracle? In reality, an infallible oracle is not available of course. What is available is a large corpus in which sixty-odd human subjects make their best stab at each of the utterance situations. We acknowledge that evaluation measures could handle the unavoidable differences between human subjects in different ways: focussing on the average Dice score of an algorithm (over all subjects as well